{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 便利店销量预测\n",
    "这是[便利店销量预测比赛](https://www.kaggle.com/c/rossmann-store-sales)的一个简单尝试参考。<br>\n",
    "by [@寒小阳](http://blog.csdn.net/han_xiaoyang)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 class=\"page-name\">\n",
    "    Forecast sales using store, promotion, and competitor data\n",
    "</h1>\n",
    "\n",
    "\n",
    "<p>Rossmann operates over 3,000 drug stores in 7 European countries. Currently, <br />Rossmann store managers are tasked with predicting their daily sales for up to six weeks in advance. Store sales are influenced by many factors, including promotions, competition, school and state holidays, seasonality, and locality. With thousands of individual managers predicting sales based on their unique circumstances, the accuracy of results can be quite varied.</p>\n",
    "<p><span style=\"font-size: 1em; line-height: 1.5em;\">In their first Kaggle competition, Rossmann is challenging you to predict 6 weeks of daily sales for 1,115 stores located across Germany. Reliable sales forecasts enable store managers to create effective staff schedules that increase productivity and motivation. By helping Rossmann create a robust prediction model, you will help store managers stay focused on what’s most important to them: their customers and their teams! </span></p>\n",
    "<p><span style=\"font-size: 1em; line-height: 1.5em;\"> <img src=\"https://kaggle2.blob.core.windows.net/competitions/kaggle/4594/media/rossmann_banner2.png\" alt=\"\" height=\"81\" width=\"640\" /><br /></span></p>\n",
    "<p><em><span style=\"font-size: 1em; line-height: 1.5em;\">If you are interested in joining Rossmann at their headquarters near Hanover, Germany, please contact Mr. Frank König (Frank.Koenig {at} rossmann.de) Rossmann is currently recruiting data scientists at <a href=\"http://www.rossmann.de/unternehmen/karriere/stellenboerse/stellenanzeigen~jid=3A5205E3-C4F9-4F5D-AA93-438D0B064D70~\">senior</a> and <a href=\"http://www.rossmann.de/unternehmen/karriere/stellenboerse/stellenanzeigen~jid=F5142F37-C823-4767-B7CF-21DE3B351D66~\">entry-level</a> positions.</span></em></p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据\n",
    "<table id=\"data-files\" class=\"nicetable full roomy align-top border\">   \n",
    "<thead>\n",
    "    <tr>\n",
    "        <th colspan=\"2\">File Name</th> \n",
    "        <th>Available Formats</th>         \n",
    "    </tr> \n",
    "</thead>\n",
    "\n",
    "    <tbody  >\n",
    "        <tr>\n",
    "\n",
    "            <td class=\"file-name\" colspan=\"2\" rowspan=\"1\">sample_submission.csv</td>\n",
    "            <td>\n",
    "<a href=\"/c/rossmann-store-sales/download/sample_submission.csv.zip\" name=\"sample_submission.csv.zip\" onclick=\"window.Intercom(&#39;trackEvent&#39;,&#39;download_compdata&#39;,{&#39;comp_id&#39;: 4594});\">.zip (55.25 kb)</a>                    </td>\n",
    "        </tr>\n",
    "\n",
    "    </tbody>\n",
    "    <tbody  >\n",
    "        <tr>\n",
    "\n",
    "            <td class=\"file-name\" colspan=\"2\" rowspan=\"1\">store.csv</td>\n",
    "            <td>\n",
    "<a href=\"/c/rossmann-store-sales/download/store.csv.zip\" name=\"store.csv.zip\" onclick=\"window.Intercom(&#39;trackEvent&#39;,&#39;download_compdata&#39;,{&#39;comp_id&#39;: 4594});\">.zip (8.33 kb)</a>                    </td>\n",
    "        </tr>\n",
    "\n",
    "    </tbody>\n",
    "    <tbody  >\n",
    "        <tr>\n",
    "\n",
    "            <td class=\"file-name\" colspan=\"2\" rowspan=\"1\">test.csv</td>\n",
    "            <td>\n",
    "<a href=\"/c/rossmann-store-sales/download/test.csv.zip\" name=\"test.csv.zip\" onclick=\"window.Intercom(&#39;trackEvent&#39;,&#39;download_compdata&#39;,{&#39;comp_id&#39;: 4594});\">.zip (143.25 kb)</a>                    </td>\n",
    "        </tr>\n",
    "\n",
    "    </tbody>\n",
    "    <tbody  >\n",
    "        <tr>\n",
    "\n",
    "            <td class=\"file-name\" colspan=\"2\" rowspan=\"1\">train.csv</td>\n",
    "            <td>\n",
    "<a href=\"/c/rossmann-store-sales/download/train.csv.zip\" name=\"train.csv.zip\" onclick=\"window.Intercom(&#39;trackEvent&#39;,&#39;download_compdata&#39;,{&#39;comp_id&#39;: 4594});\">.zip (5.66 mb)</a>                    </td>\n",
    "        </tr>\n",
    "\n",
    "    </tbody>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>You are provided with historical sales data for 1,115 Rossmann stores. The task is to forecast the \"Sales\" column for the test set. Note that some stores in the dataset were temporarily closed for refurbishment.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Files</h3>\n",
    "<ul>\n",
    "<li><strong>train.csv</strong> - historical data including Sales</li>\n",
    "<li><strong>test.csv</strong> - historical data excluding Sales</li>\n",
    "<li><strong>sample_submission.csv</strong> - a sample submission file in the correct format</li>\n",
    "<li><strong>store.csv</strong> - supplemental information about the stores</li>\n",
    "</ul>\n",
    "<h3>Data fields</h3>\n",
    "<p>Most of the fields are self-explanatory. The following are descriptions for those that aren't.</p>\n",
    "<ul>\n",
    "<li><strong>Id</strong> - an Id that represents a (Store, Date) duple within the test set</li>\n",
    "<li><strong>Store</strong> - a unique Id for each store</li>\n",
    "<li><strong>Sales</strong> - the turnover for any given day (this is what you are predicting)</li>\n",
    "<li><strong>Customers</strong> - the number of customers on a given day</li>\n",
    "<li><strong>Open</strong> - an indicator for whether the store was open: 0 = closed, 1 = open</li>\n",
    "<li><strong>StateHoliday</strong> - indicates a state holiday. Normally all stores, with few exceptions, are closed on state holidays. Note that all schools are closed on public holidays and weekends. a = public holiday, b = Easter holiday, c = Christmas, 0 = None</li>\n",
    "<li><strong>SchoolHoliday</strong> - indicates if the (Store, Date) was affected by the closure of public schools</li>\n",
    "<li><strong>StoreType</strong> - differentiates between 4 different store models: a, b, c, d</li>\n",
    "<li><strong>Assortment</strong> - describes an assortment level: a = basic, b = extra, c = extended</li>\n",
    "<li><strong>CompetitionDistance</strong> - distance in meters to the nearest competitor store</li>\n",
    "<li><strong>CompetitionOpenSince[Month/Year]</strong> - gives the approximate year and month of the time the nearest competitor was opened</li>\n",
    "<li><strong>Promo</strong> - indicates whether a store is running a promo on that day</li>\n",
    "<li><strong>Promo2</strong> - Promo2 is a continuing and consecutive promotion for some stores: 0 = store is not participating, 1 = store is participating</li>\n",
    "<li><strong>Promo2Since[Year/Week]</strong> - describes the year and calendar week when the store started participating in Promo2</li>\n",
    "<li><strong>PromoInterval</strong> - describes the consecutive intervals Promo2 is started, naming the months the promotion is started anew. E.g. \"Feb,May,Aug,Nov\" means each round starts in February, May, August, November of any given year for that store</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 引入所需的库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import datetime\n",
    "import csv\n",
    "import numpy as np\n",
    "import os\n",
    "import scipy as sp\n",
    "import xgboost as xgb\n",
    "import itertools\n",
    "import operator\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler, LabelEncoder\n",
    "from sklearn.base import TransformerMixin\n",
    "from sklearn import cross_validation\n",
    "from matplotlib import pylab as plt\n",
    "plot = True\n",
    "\n",
    "goal = 'Sales'\n",
    "myid = 'Id'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 定义一些变换和评判准则\n",
    "使用不同的loss function的时候要特别注意这个"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ToWeight(y):\n",
    "    w = np.zeros(y.shape, dtype=float)\n",
    "    ind = y != 0\n",
    "    w[ind] = 1./(y[ind]**2)\n",
    "    return w\n",
    "\n",
    "def rmspe(yhat, y):\n",
    "    w = ToWeight(y)\n",
    "    rmspe = np.sqrt(np.mean( w * (y - yhat)**2 ))\n",
    "    return rmspe\n",
    "\n",
    "def rmspe_xg(yhat, y):\n",
    "    # y = y.values\n",
    "    y = y.get_label()\n",
    "    y = np.exp(y) - 1\n",
    "    yhat = np.exp(yhat) - 1\n",
    "    w = ToWeight(y)\n",
    "    rmspe = np.sqrt(np.mean(w * (y - yhat)**2))\n",
    "    return \"rmspe\", rmspe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "store = pd.read_csv('/Users/a1/Desktop/learning/kaggle_07/Sales_prediction/store.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>StoreType</th>\n",
       "      <th>Assortment</th>\n",
       "      <th>CompetitionDistance</th>\n",
       "      <th>CompetitionOpenSinceMonth</th>\n",
       "      <th>CompetitionOpenSinceYear</th>\n",
       "      <th>Promo2</th>\n",
       "      <th>Promo2SinceWeek</th>\n",
       "      <th>Promo2SinceYear</th>\n",
       "      <th>PromoInterval</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>a</td>\n",
       "      <td>1270.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2008.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>570.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>1</td>\n",
       "      <td>13.0</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>Jan,Apr,Jul,Oct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>14130.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2006.0</td>\n",
       "      <td>1</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2011.0</td>\n",
       "      <td>Jan,Apr,Jul,Oct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>c</td>\n",
       "      <td>c</td>\n",
       "      <td>620.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2009.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>29910.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Store StoreType Assortment  CompetitionDistance  CompetitionOpenSinceMonth  \\\n",
       "0      1         c          a               1270.0                        9.0   \n",
       "1      2         a          a                570.0                       11.0   \n",
       "2      3         a          a              14130.0                       12.0   \n",
       "3      4         c          c                620.0                        9.0   \n",
       "4      5         a          a              29910.0                        4.0   \n",
       "\n",
       "   CompetitionOpenSinceYear  Promo2  Promo2SinceWeek  Promo2SinceYear  \\\n",
       "0                    2008.0       0              NaN              NaN   \n",
       "1                    2007.0       1             13.0           2010.0   \n",
       "2                    2006.0       1             14.0           2011.0   \n",
       "3                    2009.0       0              NaN              NaN   \n",
       "4                    2015.0       0              NaN              NaN   \n",
       "\n",
       "     PromoInterval  \n",
       "0              NaN  \n",
       "1  Jan,Apr,Jul,Oct  \n",
       "2  Jan,Apr,Jul,Oct  \n",
       "3              NaN  \n",
       "4              NaN  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.read_csv('/Users/a1/Desktop/learning/kaggle_07/Sales_prediction/train.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sales</th>\n",
       "      <th>Customers</th>\n",
       "      <th>Open</th>\n",
       "      <th>Promo</th>\n",
       "      <th>StateHoliday</th>\n",
       "      <th>SchoolHoliday</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>5263</td>\n",
       "      <td>555</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>6064</td>\n",
       "      <td>625</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>8314</td>\n",
       "      <td>821</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>13995</td>\n",
       "      <td>1498</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>4822</td>\n",
       "      <td>559</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Store  DayOfWeek        Date  Sales  Customers  Open  Promo StateHoliday  \\\n",
       "0      1          5  2015-07-31   5263        555     1      1            0   \n",
       "1      2          5  2015-07-31   6064        625     1      1            0   \n",
       "2      3          5  2015-07-31   8314        821     1      1            0   \n",
       "3      4          5  2015-07-31  13995       1498     1      1            0   \n",
       "4      5          5  2015-07-31   4822        559     1      1            0   \n",
       "\n",
       "   SchoolHoliday  \n",
       "0              1  \n",
       "1              1  \n",
       "2              1  \n",
       "3              1  \n",
       "4              1  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = pd.read_csv('/Users/a1/Desktop/learning/kaggle_07/Sales_prediction/test.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Store</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>Promo</th>\n",
       "      <th>StateHoliday</th>\n",
       "      <th>SchoolHoliday</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>2015-09-17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Store  DayOfWeek        Date  Open  Promo StateHoliday  SchoolHoliday\n",
       "0   1      1          4  2015-09-17   1.0      1            0              0\n",
       "1   2      3          4  2015-09-17   1.0      1            0              0\n",
       "2   3      7          4  2015-09-17   1.0      1            0              0\n",
       "3   4      8          4  2015-09-17   1.0      1            0              0\n",
       "4   5      9          4  2015-09-17   1.0      1            0              0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 加载数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def load_data():\n",
    "    \"\"\"\n",
    "        加载数据，设定数值型和非数值型数据\n",
    "    \"\"\"\n",
    "    store = pd.read_csv('./data/store.csv')\n",
    "    train_org = pd.read_csv('./data/train.csv',dtype={'StateHoliday':pd.np.string_})\n",
    "    test_org = pd.read_csv('./data/test.csv',dtype={'StateHoliday':pd.np.string_})\n",
    "    train = pd.merge(train_org,store, on='Store', how='left')\n",
    "    test = pd.merge(test_org,store, on='Store', how='left')\n",
    "    features = test.columns.tolist()\n",
    "    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n",
    "    features_numeric = test.select_dtypes(include=numerics).columns.tolist()\n",
    "    features_non_numeric = [f for f in features if f not in features_numeric]\n",
    "    return (train,test,features,features_non_numeric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据与特征处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def process_data(train,test,features,features_non_numeric):\n",
    "    \"\"\"\n",
    "        Feature engineering and selection.\n",
    "    \"\"\"\n",
    "    # # FEATURE ENGINEERING\n",
    "    train = train[train['Sales'] > 0]\n",
    "\n",
    "    for data in [train,test]:\n",
    "        # year month day\n",
    "        data['year'] = data.Date.apply(lambda x: x.split('-')[0])\n",
    "        data['year'] = data['year'].astype(float)\n",
    "        data['month'] = data.Date.apply(lambda x: x.split('-')[1])\n",
    "        data['month'] = data['month'].astype(float)\n",
    "        data['day'] = data.Date.apply(lambda x: x.split('-')[2])\n",
    "        data['day'] = data['day'].astype(float)\n",
    "\n",
    "        # promo interval \"Jan,Apr,Jul,Oct\"\n",
    "        data['promojan'] = data.PromoInterval.apply(lambda x: 0 if isinstance(x, float) else 1 if \"Jan\" in x else 0)\n",
    "        data['promofeb'] = data.PromoInterval.apply(lambda x: 0 if isinstance(x, float) else 1 if \"Feb\" in x else 0)\n",
    "        data['promomar'] = data.PromoInterval.apply(lambda x: 0 if isinstance(x, float) else 1 if \"Mar\" in x else 0)\n",
    "        data['promoapr'] = data.PromoInterval.apply(lambda x: 0 if isinstance(x, float) else 1 if \"Apr\" in x else 0)\n",
    "        data['promomay'] = data.PromoInterval.apply(lambda x: 0 if isinstance(x, float) else 1 if \"May\" in x else 0)\n",
    "        data['promojun'] = data.PromoInterval.apply(lambda x: 0 if isinstance(x, float) else 1 if \"Jun\" in x else 0)\n",
    "        data['promojul'] = data.PromoInterval.apply(lambda x: 0 if isinstance(x, float) else 1 if \"Jul\" in x else 0)\n",
    "        data['promoaug'] = data.PromoInterval.apply(lambda x: 0 if isinstance(x, float) else 1 if \"Aug\" in x else 0)\n",
    "        data['promosep'] = data.PromoInterval.apply(lambda x: 0 if isinstance(x, float) else 1 if \"Sep\" in x else 0)\n",
    "        data['promooct'] = data.PromoInterval.apply(lambda x: 0 if isinstance(x, float) else 1 if \"Oct\" in x else 0)\n",
    "        data['promonov'] = data.PromoInterval.apply(lambda x: 0 if isinstance(x, float) else 1 if \"Nov\" in x else 0)\n",
    "        data['promodec'] = data.PromoInterval.apply(lambda x: 0 if isinstance(x, float) else 1 if \"Dec\" in x else 0)\n",
    "\n",
    "    # # Features set.\n",
    "    noisy_features = [myid,'Date']\n",
    "    features = [c for c in features if c not in noisy_features]\n",
    "    features_non_numeric = [c for c in features_non_numeric if c not in noisy_features]\n",
    "    features.extend(['year','month','day'])\n",
    "    # Fill NA\n",
    "    class DataFrameImputer(TransformerMixin):\n",
    "        # http://stackoverflow.com/questions/25239958/impute-categorical-missing-values-in-scikit-learn\n",
    "        def __init__(self):\n",
    "            \"\"\"Impute missing values.\n",
    "            Columns of dtype object are imputed with the most frequent value\n",
    "            in column.\n",
    "            Columns of other types are imputed with mean of column.\n",
    "            \"\"\"\n",
    "        def fit(self, X, y=None):\n",
    "            self.fill = pd.Series([X[c].value_counts().index[0] # mode\n",
    "                if X[c].dtype == np.dtype('O') else X[c].mean() for c in X], # mean\n",
    "                index=X.columns)\n",
    "            return self\n",
    "        def transform(self, X, y=None):\n",
    "            return X.fillna(self.fill)\n",
    "    train = DataFrameImputer().fit_transform(train)\n",
    "    test = DataFrameImputer().fit_transform(test)\n",
    "    # Pre-processing non-numberic values\n",
    "    le = LabelEncoder()\n",
    "    for col in features_non_numeric:\n",
    "        le.fit(list(train[col])+list(test[col]))\n",
    "        train[col] = le.transform(train[col])\n",
    "        test[col] = le.transform(test[col])\n",
    "    # LR和神经网络这种模型都对输入数据的幅度极度敏感，请先做归一化操作\n",
    "    scaler = StandardScaler()\n",
    "    for col in set(features) - set(features_non_numeric) - \\\n",
    "      set([]): # TODO: add what not to scale\n",
    "        scaler.fit(list(train[col])+list(test[col]))\n",
    "        train[col] = scaler.transform(train[col])\n",
    "        test[col] = scaler.transform(test[col])\n",
    "    return (train,test,features,features_non_numeric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练与分析"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def XGB_native(train,test,features,features_non_numeric):\n",
    "    depth = 13\n",
    "    eta = 0.01\n",
    "    ntrees = 8000\n",
    "    mcw = 3\n",
    "    params = {\"objective\": \"reg:linear\",\n",
    "              \"booster\": \"gbtree\",\n",
    "              \"eta\": eta,\n",
    "              \"max_depth\": depth,\n",
    "              \"min_child_weight\": mcw,\n",
    "              \"subsample\": 0.9,\n",
    "              \"colsample_bytree\": 0.7,\n",
    "              \"silent\": 1\n",
    "              }\n",
    "    print \"Running with params: \" + str(params)\n",
    "    print \"Running with ntrees: \" + str(ntrees)\n",
    "    print \"Running with features: \" + str(features)\n",
    "\n",
    "    # Train model with local split\n",
    "    tsize = 0.05\n",
    "    X_train, X_test = cross_validation.train_test_split(train, test_size=tsize)\n",
    "    dtrain = xgb.DMatrix(X_train[features], np.log(X_train[goal] + 1))\n",
    "    dvalid = xgb.DMatrix(X_test[features], np.log(X_test[goal] + 1))\n",
    "    watchlist = [(dvalid, 'eval'), (dtrain, 'train')]\n",
    "    gbm = xgb.train(params, dtrain, ntrees, evals=watchlist, early_stopping_rounds=100, feval=rmspe_xg, verbose_eval=True)\n",
    "    train_probs = gbm.predict(xgb.DMatrix(X_test[features]))\n",
    "    indices = train_probs < 0\n",
    "    train_probs[indices] = 0\n",
    "    error = rmspe(np.exp(train_probs) - 1, X_test[goal].values)\n",
    "    print error\n",
    "\n",
    "    # Predict and Export\n",
    "    test_probs = gbm.predict(xgb.DMatrix(test[features]))\n",
    "    indices = test_probs < 0\n",
    "    test_probs[indices] = 0\n",
    "    submission = pd.DataFrame({myid: test[myid], goal: np.exp(test_probs) - 1})\n",
    "    if not os.path.exists('result/'):\n",
    "        os.makedirs('result/')\n",
    "    submission.to_csv(\"./result/dat-xgb_d%s_eta%s_ntree%s_mcw%s_tsize%s.csv\" % (str(depth),str(eta),str(ntrees),str(mcw),str(tsize)) , index=False)\n",
    "    # Feature importance\n",
    "    if plot:\n",
    "      outfile = open('xgb.fmap', 'w')\n",
    "      i = 0\n",
    "      for feat in features:\n",
    "          outfile.write('{0}\\t{1}\\tq\\n'.format(i, feat))\n",
    "          i = i + 1\n",
    "      outfile.close()\n",
    "      importance = gbm.get_fscore(fmap='xgb.fmap')\n",
    "      importance = sorted(importance.items(), key=operator.itemgetter(1))\n",
    "      df = pd.DataFrame(importance, columns=['feature', 'fscore'])\n",
    "      df['fscore'] = df['fscore'] / df['fscore'].sum()\n",
    "      # Plotitup\n",
    "      plt.figure()\n",
    "      df.plot()\n",
    "      df.plot(kind='barh', x='feature', y='fscore', legend=False, figsize=(25, 15))\n",
    "      plt.title('XGBoost Feature Importance')\n",
    "      plt.xlabel('relative importance')\n",
    "      plt.gcf().savefig('Feature_Importance_xgb_d%s_eta%s_ntree%s_mcw%s_tsize%s.png' % (str(depth),str(eta),str(ntrees),str(mcw),str(tsize)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=> 载入数据中...\n",
      "=> 处理数据与特征工程...\n",
      "=> 使用XGBoost建模...\n",
      "Running with params: {'subsample': 0.9, 'eta': 0.01, 'colsample_bytree': 0.7, 'silent': 1, 'objective': 'reg:linear', 'max_depth': 13, 'min_child_weight': 3, 'booster': 'gbtree'}\n",
      "Running with ntrees: 8000\n",
      "Running with features: ['Store', 'DayOfWeek', 'Open', 'Promo', 'StateHoliday', 'SchoolHoliday', 'StoreType', 'Assortment', 'CompetitionDistance', 'CompetitionOpenSinceMonth', 'CompetitionOpenSinceYear', 'Promo2', 'Promo2SinceWeek', 'Promo2SinceYear', 'PromoInterval', 'year', 'month', 'day']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Will train until train error hasn't decreased in 100 rounds.\n",
      "[0]\teval-rmspe:0.999864\ttrain-rmspe:0.999864\n",
      "[1]\teval-rmspe:0.999838\ttrain-rmspe:0.999838\n",
      "[2]\teval-rmspe:0.999810\ttrain-rmspe:0.999810\n",
      "[3]\teval-rmspe:0.999780\ttrain-rmspe:0.999780\n",
      "[4]\teval-rmspe:0.999747\ttrain-rmspe:0.999748\n",
      "[5]\teval-rmspe:0.999713\ttrain-rmspe:0.999713\n",
      "[6]\teval-rmspe:0.999676\ttrain-rmspe:0.999676\n",
      "[7]\teval-rmspe:0.999637\ttrain-rmspe:0.999637\n",
      "[8]\teval-rmspe:0.999594\ttrain-rmspe:0.999595\n",
      "[9]\teval-rmspe:0.999550\ttrain-rmspe:0.999550\n",
      "[10]\teval-rmspe:0.999502\ttrain-rmspe:0.999502\n",
      "[11]\teval-rmspe:0.999451\ttrain-rmspe:0.999451\n",
      "[12]\teval-rmspe:0.999397\ttrain-rmspe:0.999397\n",
      "[13]\teval-rmspe:0.999339\ttrain-rmspe:0.999339\n",
      "[14]\teval-rmspe:0.999277\ttrain-rmspe:0.999278\n",
      "[15]\teval-rmspe:0.999212\ttrain-rmspe:0.999213\n",
      "[16]\teval-rmspe:0.999143\ttrain-rmspe:0.999143\n",
      "[17]\teval-rmspe:0.999069\ttrain-rmspe:0.999070\n",
      "[18]\teval-rmspe:0.998991\ttrain-rmspe:0.998992\n",
      "[19]\teval-rmspe:0.998908\ttrain-rmspe:0.998909\n",
      "[20]\teval-rmspe:0.998821\ttrain-rmspe:0.998822\n",
      "[21]\teval-rmspe:0.998728\ttrain-rmspe:0.998729\n",
      "[22]\teval-rmspe:0.998630\ttrain-rmspe:0.998631\n",
      "[23]\teval-rmspe:0.998527\ttrain-rmspe:0.998528\n",
      "[24]\teval-rmspe:0.998418\ttrain-rmspe:0.998418\n",
      "[25]\teval-rmspe:0.998303\ttrain-rmspe:0.998303\n",
      "[26]\teval-rmspe:0.998181\ttrain-rmspe:0.998182\n",
      "[27]\teval-rmspe:0.998053\ttrain-rmspe:0.998053\n",
      "[28]\teval-rmspe:0.997917\ttrain-rmspe:0.997918\n",
      "[29]\teval-rmspe:0.997775\ttrain-rmspe:0.997776\n",
      "[30]\teval-rmspe:0.997626\ttrain-rmspe:0.997627\n",
      "[31]\teval-rmspe:0.997469\ttrain-rmspe:0.997470\n",
      "[32]\teval-rmspe:0.997303\ttrain-rmspe:0.997304\n",
      "[33]\teval-rmspe:0.997130\ttrain-rmspe:0.997130\n",
      "[34]\teval-rmspe:0.996947\ttrain-rmspe:0.996948\n",
      "[35]\teval-rmspe:0.996756\ttrain-rmspe:0.996757\n",
      "[36]\teval-rmspe:0.996556\ttrain-rmspe:0.996557\n",
      "[37]\teval-rmspe:0.996347\ttrain-rmspe:0.996348\n",
      "[38]\teval-rmspe:0.996127\ttrain-rmspe:0.996128\n",
      "[39]\teval-rmspe:0.995897\ttrain-rmspe:0.995898\n",
      "[40]\teval-rmspe:0.995657\ttrain-rmspe:0.995658\n",
      "[41]\teval-rmspe:0.995405\ttrain-rmspe:0.995406\n",
      "[42]\teval-rmspe:0.995143\ttrain-rmspe:0.995143\n",
      "[43]\teval-rmspe:0.994869\ttrain-rmspe:0.994870\n",
      "[44]\teval-rmspe:0.994582\ttrain-rmspe:0.994583\n",
      "[45]\teval-rmspe:0.994284\ttrain-rmspe:0.994285\n",
      "[46]\teval-rmspe:0.993972\ttrain-rmspe:0.993973\n",
      "[47]\teval-rmspe:0.993647\ttrain-rmspe:0.993648\n",
      "[48]\teval-rmspe:0.993310\ttrain-rmspe:0.993310\n",
      "[49]\teval-rmspe:0.992957\ttrain-rmspe:0.992958\n",
      "[50]\teval-rmspe:0.992592\ttrain-rmspe:0.992592\n",
      "[51]\teval-rmspe:0.992212\ttrain-rmspe:0.992212\n",
      "[52]\teval-rmspe:0.991815\ttrain-rmspe:0.991815\n",
      "[53]\teval-rmspe:0.991404\ttrain-rmspe:0.991404\n",
      "[54]\teval-rmspe:0.990975\ttrain-rmspe:0.990975\n",
      "[55]\teval-rmspe:0.990532\ttrain-rmspe:0.990532\n",
      "[56]\teval-rmspe:0.990072\ttrain-rmspe:0.990072\n",
      "[57]\teval-rmspe:0.989596\ttrain-rmspe:0.989595\n",
      "[58]\teval-rmspe:0.989102\ttrain-rmspe:0.989101\n",
      "[59]\teval-rmspe:0.988589\ttrain-rmspe:0.988588\n",
      "[60]\teval-rmspe:0.988058\ttrain-rmspe:0.988058\n",
      "[61]\teval-rmspe:0.987509\ttrain-rmspe:0.987508\n",
      "[62]\teval-rmspe:0.986942\ttrain-rmspe:0.986941\n",
      "[63]\teval-rmspe:0.986356\ttrain-rmspe:0.986355\n",
      "[64]\teval-rmspe:0.985748\ttrain-rmspe:0.985747\n",
      "[65]\teval-rmspe:0.985122\ttrain-rmspe:0.985121\n",
      "[66]\teval-rmspe:0.984474\ttrain-rmspe:0.984473\n",
      "[67]\teval-rmspe:0.983806\ttrain-rmspe:0.983804\n",
      "[68]\teval-rmspe:0.983117\ttrain-rmspe:0.983115\n",
      "[69]\teval-rmspe:0.982407\ttrain-rmspe:0.982406\n",
      "[70]\teval-rmspe:0.981673\ttrain-rmspe:0.981671\n",
      "[71]\teval-rmspe:0.980916\ttrain-rmspe:0.980914\n",
      "[72]\teval-rmspe:0.980138\ttrain-rmspe:0.980136\n",
      "[73]\teval-rmspe:0.979336\ttrain-rmspe:0.979333\n",
      "[74]\teval-rmspe:0.978509\ttrain-rmspe:0.978506\n",
      "[75]\teval-rmspe:0.977662\ttrain-rmspe:0.977660\n",
      "[76]\teval-rmspe:0.976788\ttrain-rmspe:0.976786\n",
      "[77]\teval-rmspe:0.975891\ttrain-rmspe:0.975887\n",
      "[78]\teval-rmspe:0.974967\ttrain-rmspe:0.974963\n",
      "[79]\teval-rmspe:0.974015\ttrain-rmspe:0.974011\n",
      "[80]\teval-rmspe:0.973041\ttrain-rmspe:0.973037\n",
      "[81]\teval-rmspe:0.972039\ttrain-rmspe:0.972036\n",
      "[82]\teval-rmspe:0.971013\ttrain-rmspe:0.971010\n",
      "[83]\teval-rmspe:0.969961\ttrain-rmspe:0.969957\n",
      "[84]\teval-rmspe:0.968882\ttrain-rmspe:0.968878\n",
      "[85]\teval-rmspe:0.967775\ttrain-rmspe:0.967770\n",
      "[86]\teval-rmspe:0.966639\ttrain-rmspe:0.966635\n",
      "[87]\teval-rmspe:0.965480\ttrain-rmspe:0.965475\n",
      "[88]\teval-rmspe:0.964287\ttrain-rmspe:0.964283\n",
      "[89]\teval-rmspe:0.963068\ttrain-rmspe:0.963063\n",
      "[90]\teval-rmspe:0.961818\ttrain-rmspe:0.961814\n",
      "[91]\teval-rmspe:0.960541\ttrain-rmspe:0.960536\n",
      "[92]\teval-rmspe:0.959236\ttrain-rmspe:0.959231\n",
      "[93]\teval-rmspe:0.957903\ttrain-rmspe:0.957898\n",
      "[94]\teval-rmspe:0.956536\ttrain-rmspe:0.956532\n",
      "[95]\teval-rmspe:0.955146\ttrain-rmspe:0.955141\n",
      "[96]\teval-rmspe:0.953722\ttrain-rmspe:0.953717\n",
      "[97]\teval-rmspe:0.952264\ttrain-rmspe:0.952260\n",
      "[98]\teval-rmspe:0.950781\ttrain-rmspe:0.950777\n",
      "[99]\teval-rmspe:0.949263\ttrain-rmspe:0.949260\n",
      "[100]\teval-rmspe:0.947724\ttrain-rmspe:0.947721\n",
      "[101]\teval-rmspe:0.946153\ttrain-rmspe:0.946150\n",
      "[102]\teval-rmspe:0.944546\ttrain-rmspe:0.944544\n",
      "[103]\teval-rmspe:0.942909\ttrain-rmspe:0.942907\n",
      "[104]\teval-rmspe:0.941239\ttrain-rmspe:0.941238\n",
      "[105]\teval-rmspe:0.939537\ttrain-rmspe:0.939537\n",
      "[106]\teval-rmspe:0.937813\ttrain-rmspe:0.937813\n",
      "[107]\teval-rmspe:0.936048\ttrain-rmspe:0.936050\n",
      "[108]\teval-rmspe:0.934255\ttrain-rmspe:0.934258\n",
      "[109]\teval-rmspe:0.932432\ttrain-rmspe:0.932436\n",
      "[110]\teval-rmspe:0.930573\ttrain-rmspe:0.930578\n",
      "[111]\teval-rmspe:0.928691\ttrain-rmspe:0.928697\n",
      "[112]\teval-rmspe:0.926783\ttrain-rmspe:0.926790\n",
      "[113]\teval-rmspe:0.924841\ttrain-rmspe:0.924849\n",
      "[114]\teval-rmspe:0.922866\ttrain-rmspe:0.922875\n",
      "[115]\teval-rmspe:0.920861\ttrain-rmspe:0.920872\n",
      "[116]\teval-rmspe:0.918817\ttrain-rmspe:0.918830\n",
      "[117]\teval-rmspe:0.916740\ttrain-rmspe:0.916756\n",
      "[118]\teval-rmspe:0.914641\ttrain-rmspe:0.914659\n",
      "[119]\teval-rmspe:0.912504\ttrain-rmspe:0.912524\n",
      "[120]\teval-rmspe:0.910340\ttrain-rmspe:0.910363\n",
      "[121]\teval-rmspe:0.908146\ttrain-rmspe:0.908172\n",
      "[122]\teval-rmspe:0.905924\ttrain-rmspe:0.905952\n",
      "[123]\teval-rmspe:0.903665\ttrain-rmspe:0.903696\n",
      "[124]\teval-rmspe:0.901385\ttrain-rmspe:0.901419\n",
      "[125]\teval-rmspe:0.899064\ttrain-rmspe:0.899100\n",
      "[126]\teval-rmspe:0.896714\ttrain-rmspe:0.896753\n",
      "[127]\teval-rmspe:0.894339\ttrain-rmspe:0.894382\n",
      "[128]\teval-rmspe:0.891930\ttrain-rmspe:0.891975\n",
      "[129]\teval-rmspe:0.889488\ttrain-rmspe:0.889537\n",
      "[130]\teval-rmspe:0.887016\ttrain-rmspe:0.887070\n",
      "[131]\teval-rmspe:0.884516\ttrain-rmspe:0.884575\n",
      "[132]\teval-rmspe:0.881991\ttrain-rmspe:0.882054\n",
      "[133]\teval-rmspe:0.879438\ttrain-rmspe:0.879505\n",
      "[134]\teval-rmspe:0.876871\ttrain-rmspe:0.876942\n",
      "[135]\teval-rmspe:0.874260\ttrain-rmspe:0.874337\n",
      "[136]\teval-rmspe:0.871616\ttrain-rmspe:0.871698\n",
      "[137]\teval-rmspe:0.868961\ttrain-rmspe:0.869046\n",
      "[138]\teval-rmspe:0.866260\ttrain-rmspe:0.866351\n",
      "[139]\teval-rmspe:0.863534\ttrain-rmspe:0.863632\n",
      "[140]\teval-rmspe:0.860796\ttrain-rmspe:0.860899\n",
      "[141]\teval-rmspe:0.858021\ttrain-rmspe:0.858131\n",
      "[142]\teval-rmspe:0.855213\ttrain-rmspe:0.855331\n",
      "[143]\teval-rmspe:0.852387\ttrain-rmspe:0.852512\n",
      "[144]\teval-rmspe:0.849545\ttrain-rmspe:0.849678\n",
      "[145]\teval-rmspe:0.846674\ttrain-rmspe:0.846814\n",
      "[146]\teval-rmspe:0.843768\ttrain-rmspe:0.843917\n",
      "[147]\teval-rmspe:0.840848\ttrain-rmspe:0.841005\n",
      "[148]\teval-rmspe:0.837907\ttrain-rmspe:0.838073\n",
      "[149]\teval-rmspe:0.834927\ttrain-rmspe:0.835103\n",
      "[150]\teval-rmspe:0.831933\ttrain-rmspe:0.832115\n",
      "[151]\teval-rmspe:0.828913\ttrain-rmspe:0.829105\n",
      "[152]\teval-rmspe:0.825874\ttrain-rmspe:0.826077\n",
      "[153]\teval-rmspe:0.822809\ttrain-rmspe:0.823022\n",
      "[154]\teval-rmspe:0.819724\ttrain-rmspe:0.819949\n",
      "[155]\teval-rmspe:0.816622\ttrain-rmspe:0.816859\n",
      "[156]\teval-rmspe:0.813492\ttrain-rmspe:0.813740\n",
      "[157]\teval-rmspe:0.810347\ttrain-rmspe:0.810608\n",
      "[158]\teval-rmspe:0.807189\ttrain-rmspe:0.807463\n",
      "[159]\teval-rmspe:0.804018\ttrain-rmspe:0.804305\n",
      "[160]\teval-rmspe:0.800810\ttrain-rmspe:0.801112\n",
      "[161]\teval-rmspe:0.797586\ttrain-rmspe:0.797902\n",
      "[162]\teval-rmspe:0.794338\ttrain-rmspe:0.794668\n",
      "[163]\teval-rmspe:0.791072\ttrain-rmspe:0.791419\n",
      "[164]\teval-rmspe:0.787807\ttrain-rmspe:0.788168\n",
      "[165]\teval-rmspe:0.784528\ttrain-rmspe:0.784904\n",
      "[166]\teval-rmspe:0.781223\ttrain-rmspe:0.781614\n",
      "[167]\teval-rmspe:0.777907\ttrain-rmspe:0.778315\n",
      "[168]\teval-rmspe:0.774574\ttrain-rmspe:0.774998\n",
      "[169]\teval-rmspe:0.771232\ttrain-rmspe:0.771673\n",
      "[170]\teval-rmspe:0.767850\ttrain-rmspe:0.768309\n",
      "[171]\teval-rmspe:0.764459\ttrain-rmspe:0.764921\n",
      "[172]\teval-rmspe:0.761063\ttrain-rmspe:0.761544\n",
      "[173]\teval-rmspe:0.757638\ttrain-rmspe:0.758139\n",
      "[174]\teval-rmspe:0.754222\ttrain-rmspe:0.754743\n",
      "[175]\teval-rmspe:0.750783\ttrain-rmspe:0.751324\n",
      "[176]\teval-rmspe:0.747327\ttrain-rmspe:0.747891\n",
      "[177]\teval-rmspe:0.743874\ttrain-rmspe:0.744462\n",
      "[178]\teval-rmspe:0.740393\ttrain-rmspe:0.741004\n",
      "[179]\teval-rmspe:0.736912\ttrain-rmspe:0.737547\n",
      "[180]\teval-rmspe:0.733413\ttrain-rmspe:0.734073\n",
      "[181]\teval-rmspe:0.729911\ttrain-rmspe:0.730592\n",
      "[182]\teval-rmspe:0.726400\ttrain-rmspe:0.727108\n",
      "[183]\teval-rmspe:0.722867\ttrain-rmspe:0.723602\n",
      "[184]\teval-rmspe:0.719336\ttrain-rmspe:0.720100\n",
      "[185]\teval-rmspe:0.715804\ttrain-rmspe:0.716595\n",
      "[186]\teval-rmspe:0.712266\ttrain-rmspe:0.713081\n",
      "[187]\teval-rmspe:0.708727\ttrain-rmspe:0.709572\n",
      "[188]\teval-rmspe:0.705180\ttrain-rmspe:0.706057\n",
      "[189]\teval-rmspe:0.701624\ttrain-rmspe:0.702531\n",
      "[190]\teval-rmspe:0.698065\ttrain-rmspe:0.699002\n",
      "[191]\teval-rmspe:0.694486\ttrain-rmspe:0.695456\n",
      "[192]\teval-rmspe:0.690903\ttrain-rmspe:0.691906\n",
      "[193]\teval-rmspe:0.687327\ttrain-rmspe:0.688363\n",
      "[194]\teval-rmspe:0.683744\ttrain-rmspe:0.684816\n",
      "[195]\teval-rmspe:0.680163\ttrain-rmspe:0.681267\n",
      "[196]\teval-rmspe:0.676581\ttrain-rmspe:0.677718\n",
      "[197]\teval-rmspe:0.672995\ttrain-rmspe:0.674167\n",
      "[198]\teval-rmspe:0.669403\ttrain-rmspe:0.670615\n",
      "[199]\teval-rmspe:0.665808\ttrain-rmspe:0.667054\n",
      "[200]\teval-rmspe:0.662204\ttrain-rmspe:0.663490\n",
      "[201]\teval-rmspe:0.658612\ttrain-rmspe:0.659936\n",
      "[202]\teval-rmspe:0.655009\ttrain-rmspe:0.656374\n",
      "[203]\teval-rmspe:0.651414\ttrain-rmspe:0.652821\n",
      "[204]\teval-rmspe:0.647823\ttrain-rmspe:0.649273\n",
      "[205]\teval-rmspe:0.644224\ttrain-rmspe:0.645712\n",
      "[206]\teval-rmspe:0.640633\ttrain-rmspe:0.642163\n",
      "[207]\teval-rmspe:0.637032\ttrain-rmspe:0.638610\n",
      "[208]\teval-rmspe:0.633434\ttrain-rmspe:0.635059\n",
      "[209]\teval-rmspe:0.629856\ttrain-rmspe:0.631529\n",
      "[210]\teval-rmspe:0.626272\ttrain-rmspe:0.627993\n",
      "[211]\teval-rmspe:0.622688\ttrain-rmspe:0.624438\n",
      "[212]\teval-rmspe:0.619104\ttrain-rmspe:0.620901\n",
      "[213]\teval-rmspe:0.615533\ttrain-rmspe:0.617381\n",
      "[214]\teval-rmspe:0.611960\ttrain-rmspe:0.613804\n",
      "[215]\teval-rmspe:0.608390\ttrain-rmspe:0.610288\n",
      "[216]\teval-rmspe:0.604828\ttrain-rmspe:0.606736\n",
      "[217]\teval-rmspe:0.601281\ttrain-rmspe:0.603231\n",
      "[218]\teval-rmspe:0.597736\ttrain-rmspe:0.599731\n",
      "[219]\teval-rmspe:0.594199\ttrain-rmspe:0.596253\n",
      "[220]\teval-rmspe:0.590670\ttrain-rmspe:0.592772\n",
      "[221]\teval-rmspe:0.587147\ttrain-rmspe:0.589304\n",
      "[222]\teval-rmspe:0.583627\ttrain-rmspe:0.585833\n",
      "[223]\teval-rmspe:0.580118\ttrain-rmspe:0.582381\n",
      "[224]\teval-rmspe:0.576615\ttrain-rmspe:0.578939\n",
      "[225]\teval-rmspe:0.573117\ttrain-rmspe:0.575501\n",
      "[226]\teval-rmspe:0.569634\ttrain-rmspe:0.572083\n",
      "[227]\teval-rmspe:0.566160\ttrain-rmspe:0.568666\n",
      "[228]\teval-rmspe:0.562700\ttrain-rmspe:0.565268\n",
      "[229]\teval-rmspe:0.559248\ttrain-rmspe:0.561880\n",
      "[230]\teval-rmspe:0.555806\ttrain-rmspe:0.558472\n",
      "[231]\teval-rmspe:0.552370\ttrain-rmspe:0.555089\n",
      "[232]\teval-rmspe:0.548946\ttrain-rmspe:0.551725\n",
      "[233]\teval-rmspe:0.545530\ttrain-rmspe:0.548373\n",
      "[234]\teval-rmspe:0.542128\ttrain-rmspe:0.544979\n",
      "[235]\teval-rmspe:0.538738\ttrain-rmspe:0.541658\n",
      "[236]\teval-rmspe:0.535359\ttrain-rmspe:0.538285\n",
      "[237]\teval-rmspe:0.531994\ttrain-rmspe:0.534954\n",
      "[238]\teval-rmspe:0.528642\ttrain-rmspe:0.531675\n",
      "[239]\teval-rmspe:0.525300\ttrain-rmspe:0.528395\n",
      "[240]\teval-rmspe:0.521970\ttrain-rmspe:0.525119\n",
      "[241]\teval-rmspe:0.518657\ttrain-rmspe:0.521875\n",
      "[242]\teval-rmspe:0.515360\ttrain-rmspe:0.518650\n",
      "[243]\teval-rmspe:0.512069\ttrain-rmspe:0.515436\n",
      "[244]\teval-rmspe:0.508801\ttrain-rmspe:0.512158\n",
      "[245]\teval-rmspe:0.505542\ttrain-rmspe:0.508967\n",
      "[246]\teval-rmspe:0.502301\ttrain-rmspe:0.505802\n",
      "[247]\teval-rmspe:0.499072\ttrain-rmspe:0.502568\n",
      "[248]\teval-rmspe:0.495861\ttrain-rmspe:0.499409\n",
      "[249]\teval-rmspe:0.492660\ttrain-rmspe:0.496207\n",
      "[250]\teval-rmspe:0.489473\ttrain-rmspe:0.493101\n",
      "[251]\teval-rmspe:0.486301\ttrain-rmspe:0.489997\n",
      "[252]\teval-rmspe:0.483143\ttrain-rmspe:0.486908\n",
      "[253]\teval-rmspe:0.480005\ttrain-rmspe:0.483849\n",
      "[254]\teval-rmspe:0.476886\ttrain-rmspe:0.480812\n",
      "[255]\teval-rmspe:0.473782\ttrain-rmspe:0.477789\n",
      "[256]\teval-rmspe:0.470686\ttrain-rmspe:0.474778\n",
      "[257]\teval-rmspe:0.467605\ttrain-rmspe:0.471782\n",
      "[258]\teval-rmspe:0.464537\ttrain-rmspe:0.468798\n",
      "[259]\teval-rmspe:0.461504\ttrain-rmspe:0.465849\n",
      "[260]\teval-rmspe:0.458486\ttrain-rmspe:0.462883\n",
      "[261]\teval-rmspe:0.455457\ttrain-rmspe:0.459945\n",
      "[262]\teval-rmspe:0.452470\ttrain-rmspe:0.457031\n",
      "[263]\teval-rmspe:0.449508\ttrain-rmspe:0.454157\n",
      "[264]\teval-rmspe:0.446550\ttrain-rmspe:0.451285\n",
      "[265]\teval-rmspe:0.443623\ttrain-rmspe:0.448448\n",
      "[266]\teval-rmspe:0.440713\ttrain-rmspe:0.445626\n",
      "[267]\teval-rmspe:0.437818\ttrain-rmspe:0.442742\n",
      "[268]\teval-rmspe:0.434943\ttrain-rmspe:0.439899\n",
      "[269]\teval-rmspe:0.432090\ttrain-rmspe:0.437131\n",
      "[270]\teval-rmspe:0.429257\ttrain-rmspe:0.434280\n",
      "[271]\teval-rmspe:0.426442\ttrain-rmspe:0.431552\n",
      "[272]\teval-rmspe:0.423644\ttrain-rmspe:0.428818\n",
      "[273]\teval-rmspe:0.420857\ttrain-rmspe:0.426133\n",
      "[274]\teval-rmspe:0.418090\ttrain-rmspe:0.423427\n",
      "[275]\teval-rmspe:0.415340\ttrain-rmspe:0.420774\n",
      "[276]\teval-rmspe:0.412621\ttrain-rmspe:0.418084\n",
      "[277]\teval-rmspe:0.409919\ttrain-rmspe:0.415451\n",
      "[278]\teval-rmspe:0.407238\ttrain-rmspe:0.412865\n",
      "[279]\teval-rmspe:0.404579\ttrain-rmspe:0.410286\n",
      "[280]\teval-rmspe:0.401938\ttrain-rmspe:0.407717\n",
      "[281]\teval-rmspe:0.399287\ttrain-rmspe:0.405173\n",
      "[282]\teval-rmspe:0.396676\ttrain-rmspe:0.402659\n",
      "[283]\teval-rmspe:0.394083\ttrain-rmspe:0.400166\n",
      "[284]\teval-rmspe:0.391519\ttrain-rmspe:0.397708\n",
      "[285]\teval-rmspe:0.388986\ttrain-rmspe:0.395199\n",
      "[286]\teval-rmspe:0.386444\ttrain-rmspe:0.392762\n",
      "[287]\teval-rmspe:0.383928\ttrain-rmspe:0.390350\n",
      "[288]\teval-rmspe:0.381446\ttrain-rmspe:0.387937\n",
      "[289]\teval-rmspe:0.378978\ttrain-rmspe:0.385567\n",
      "[290]\teval-rmspe:0.376485\ttrain-rmspe:0.383176\n",
      "[291]\teval-rmspe:0.374064\ttrain-rmspe:0.380707\n",
      "[292]\teval-rmspe:0.371648\ttrain-rmspe:0.378380\n",
      "[293]\teval-rmspe:0.369268\ttrain-rmspe:0.375973\n",
      "[294]\teval-rmspe:0.366876\ttrain-rmspe:0.373680\n",
      "[295]\teval-rmspe:0.364530\ttrain-rmspe:0.371400\n",
      "[296]\teval-rmspe:0.362199\ttrain-rmspe:0.369171\n",
      "[297]\teval-rmspe:0.359884\ttrain-rmspe:0.366944\n",
      "[298]\teval-rmspe:0.357610\ttrain-rmspe:0.364654\n",
      "[299]\teval-rmspe:0.355357\ttrain-rmspe:0.362500\n",
      "[300]\teval-rmspe:0.353066\ttrain-rmspe:0.360315\n",
      "[301]\teval-rmspe:0.350850\ttrain-rmspe:0.358120\n",
      "[302]\teval-rmspe:0.348654\ttrain-rmspe:0.355849\n",
      "[303]\teval-rmspe:0.346417\ttrain-rmspe:0.353714\n",
      "[304]\teval-rmspe:0.344271\ttrain-rmspe:0.351512\n",
      "[305]\teval-rmspe:0.342142\ttrain-rmspe:0.349361\n",
      "[306]\teval-rmspe:0.339941\ttrain-rmspe:0.347270\n",
      "[307]\teval-rmspe:0.337833\ttrain-rmspe:0.345234\n",
      "[308]\teval-rmspe:0.335749\ttrain-rmspe:0.343239\n",
      "[309]\teval-rmspe:0.333685\ttrain-rmspe:0.341267\n",
      "[310]\teval-rmspe:0.331629\ttrain-rmspe:0.339320\n",
      "[311]\teval-rmspe:0.329614\ttrain-rmspe:0.337300\n",
      "[312]\teval-rmspe:0.327619\ttrain-rmspe:0.335254\n",
      "[313]\teval-rmspe:0.325638\ttrain-rmspe:0.333346\n",
      "[314]\teval-rmspe:0.323649\ttrain-rmspe:0.331452\n",
      "[315]\teval-rmspe:0.321682\ttrain-rmspe:0.329566\n",
      "[316]\teval-rmspe:0.319734\ttrain-rmspe:0.327706\n",
      "[317]\teval-rmspe:0.317831\ttrain-rmspe:0.325903\n",
      "[318]\teval-rmspe:0.315936\ttrain-rmspe:0.324120\n",
      "[319]\teval-rmspe:0.314067\ttrain-rmspe:0.322258\n",
      "[320]\teval-rmspe:0.312201\ttrain-rmspe:0.320489\n",
      "[321]\teval-rmspe:0.310397\ttrain-rmspe:0.318799\n",
      "[322]\teval-rmspe:0.308579\ttrain-rmspe:0.317072\n",
      "[323]\teval-rmspe:0.306776\ttrain-rmspe:0.315366\n",
      "[324]\teval-rmspe:0.305025\ttrain-rmspe:0.313709\n",
      "[325]\teval-rmspe:0.303214\ttrain-rmspe:0.311992\n",
      "[326]\teval-rmspe:0.301484\ttrain-rmspe:0.310345\n",
      "[327]\teval-rmspe:0.299785\ttrain-rmspe:0.308697\n",
      "[328]\teval-rmspe:0.298099\ttrain-rmspe:0.307095\n",
      "[329]\teval-rmspe:0.296425\ttrain-rmspe:0.305521\n",
      "[330]\teval-rmspe:0.294744\ttrain-rmspe:0.303948\n",
      "[331]\teval-rmspe:0.293088\ttrain-rmspe:0.302402\n",
      "[332]\teval-rmspe:0.291456\ttrain-rmspe:0.300756\n",
      "[333]\teval-rmspe:0.289895\ttrain-rmspe:0.299292\n",
      "[334]\teval-rmspe:0.288335\ttrain-rmspe:0.297665\n",
      "[335]\teval-rmspe:0.286746\ttrain-rmspe:0.296190\n"
     ]
    }
   ],
   "source": [
    "print \"=> 载入数据中...\"\n",
    "train,test,features,features_non_numeric = load_data()\n",
    "print \"=> 处理数据与特征工程...\"\n",
    "train,test,features,features_non_numeric = process_data(train,test,features,features_non_numeric)\n",
    "print \"=> 使用XGBoost建模...\"\n",
    "XGB_native(train,test,features,features_non_numeric)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
